EDA on raw data (after collect)¶

This notebook main goal is to :

Understand what is in the data : plot variables one by one, missing values, etc.
See which data are correlated
Removes useless variables : too much missing values, too correlated with other variables.
See if data is biased on protected attributes

Load packages¶

import pandas as pd
import numpy as np

import matplotlib.pyplot as plt
import seaborn as sns
from IPython.display import display, Markdown

# transparentai package : https://github.com/Nathanlauga/transparentai
from transparentai.datasets import ClassificationDataset
import transparentai.explore as explore

PROJECT_PATH = '..'

Load data informations¶

df_details = pd.read_csv(f'{PROJECT_PATH}/01_collect/columns_informations.csv')

df_details

Load data using data informations¶

dfs = {}
datasets = df_details['dataset'].unique()

for dataset in datasets:
    dataset_detail = df_details[df_details['dataset'] == dataset]

    dtypes, columns = dataset_detail['dtype'], dataset_detail['column']
    dtypes = pd.Series(dtypes.values,index=columns)

    parse_dates = list()
    for var, dtype in dtypes.iteritems():
        if 'datetime' in dtype:
            parse_dates.append(var)

    dtypes = dtypes.str.replace(r'datetime.*','str')

    dfs[dataset] = pd.read_csv(f'{PROJECT_PATH}/_data/{dataset}.csv',
                               dtype=dtypes.to_dict(),
                               parse_dates=parse_dates
                              )

    columns = np.where(dataset_detail['new_name'].isna(), columns, dataset_detail['new_name'])
    dfs[dataset].columns = columns

Display shape & head¶

for dataset in dfs:
    display(Markdown(f'#### {dataset}, {dfs[dataset].shape}'))
    display(dfs[dataset].head())

Set datasets into differents variables¶

Here we just have one dataset adult, but if in the next versions we had some others datasets it's important to set them into distinct variables.

adult = dfs['adult']
del dfs

Analyse : missing values¶

help(explore.show_missing_values)

Help on function show_missing_values in module transparentai.explore.explore:

show_missing_values(df)
    Show a bar plot that display percentage of missing values on columns that have some.
    If no missing value then it use `display` & `Markdown` functions to indicate it.
    
    Parameters
    ----------
    df: pd.DataFrame
        Dataframe to inspect

display(Markdown('#### Missing values for adult dataset'))
explore.show_missing_values(adult)

Analyse : plot each variable¶

Using MLHelper¶

from transparentai.utils import remove_var_with_one_value
adult = remove_var_with_one_value(adult)

help(explore.show_df_vars)

Help on function show_df_vars in module transparentai.explore.explore:

show_df_vars(df, target=None)
    Show all variables with graphics to understand each variable.
    If target is set, complement visuals will be added to take a look on the
    influence that a variable can have on target
    
    Data type handle : categorical, numerical, datetime
    
    Parameters
    ----------
    df: pd.DataFrame
        Dataframe to inspect
    target: str (optional)
        Target column for classifier

explore.show_df_vars(df=adult, target='income')

help(explore.show_df_numerical_relations)

Help on function show_df_numerical_relations in module transparentai.explore.explore:

show_df_numerical_relations(df, target=None)
    Show all numerical variables 2 by 2 with graphics understand their relation.
    If target is set, separate dataset for each target value.
    
    Parameters
    ----------
    df: pd.DataFrame
        Dataframe to inspect
    target: str (optional)
        Target column for classifier

explore.show_df_numerical_relations(df=adult, target='income')

help(explore.show_df_num_cat_relations)

Help on function show_df_num_cat_relations in module transparentai.explore.explore:

show_df_num_cat_relations(df, target=None)
    Show boxplots for each pair of categorical and numerical variables
    If target is set, separate dataset for each target value.
    
    Parameters
    ----------
    df: pd.DataFrame
        Dataframe to inspect
    target: str (optional)
        Target column for classifier

explore.show_df_num_cat_relations(df=adult, target='income')

help(explore.show_df_correlations)

Help on function show_df_correlations in module transparentai.explore.explore:

show_df_correlations(df)
    Show differents correlations matrix for 3 cases :
    - numerical to numerical (using Pearson coeff)
    - categorical to categorical (using Cramers V & Chi square)
    - numerical to categorical (discrete) (using Point Biserial)
    
    Parameters
    ----------
    df: pd.DataFrame
        Dataframe to inspect

explore.show_df_correlations(df=adult)

Handle dataset bias¶

Now let's take a look on dataset bias.

protected_vars = df_details[df_details['is_protected'] == 1]
protected_vars = np.where(protected_vars['new_name'].isna(), protected_vars['column'], protected_vars['new_name'])

for var in protected_vars:
    display(Markdown(f'#### {var}'))
    display(adult[var].unique())

array([25, 38, 28, 44, 18, 34, 29, 63, 24, 55, 65, 36, 26, 58, 48, 43, 20,
       37, 40, 72, 45, 22, 23, 54, 32, 46, 56, 17, 39, 52, 21, 42, 33, 30,
       47, 41, 19, 69, 50, 31, 59, 49, 51, 27, 57, 61, 64, 79, 73, 53, 77,
       80, 62, 35, 68, 66, 75, 60, 67, 71, 70, 90, 81, 74, 78, 82, 83, 85,
       76, 84, 89, 88, 87, 86])

array(['Never-married', 'Married-civ-spouse', 'Widowed', 'Divorced',
       'Separated', 'Married-spouse-absent', 'Married-AF-spouse'],
      dtype=object)

array(['Black', 'White', 'Asian-Pac-Islander', 'Other',
       'Amer-Indian-Eskimo'], dtype=object)

array(['Male', 'Female'], dtype=object)

array(['United-States', '?', 'Peru', 'Guatemala', 'Mexico',
       'Dominican-Republic', 'Ireland', 'Germany', 'Philippines',
       'Thailand', 'Haiti', 'El-Salvador', 'Puerto-Rico', 'Vietnam',
       'South', 'Columbia', 'Japan', 'India', 'Cambodia', 'Poland',
       'Laos', 'England', 'Cuba', 'Taiwan', 'Italy', 'Canada', 'Portugal',
       'China', 'Nicaragua', 'Honduras', 'Iran', 'Scotland', 'Jamaica',
       'Ecuador', 'Yugoslavia', 'Hungary', 'Hong', 'Greece',
       'Trinadad&Tobago', 'Outlying-US(Guam-USVI-etc)', 'France',
       'Holand-Netherlands'], dtype=object)

Convert age into a categorical variable for this purpose with the following rule :

if age < 26 then Young
else if age < 61 then Adult
else Elder

adult['age category'] = np.where(adult['age'] < 26, 'Young',
                                 np.where(adult['age'] < 61, 'Adult','Elder'))

Using transparentai ClassificationDataset dataset¶

target = 'income'
privileged_values = {
    'age category': ['Adult'],
    'marital-status': ['Married-civ-spouse','Married-AF-spouse'],
    'race': ['White'],
    'gender': ['Male']
}
fair_dataset = ClassificationDataset(df=adult,
                                     label_name=target,
                                     privileged_values=privileged_values)

fair_dataset.show_bias_metrics(label_value='>50K')

from ipywidgets import interact
import ipywidgets as widgets

attrs = list(fair_dataset.protected_attributes.keys())
labels = fair_dataset.df[fair_dataset.label_name].values.tolist()

attrs = widgets.Dropdown(options=attrs, value=attrs[0])
labels = widgets.Dropdown(options=labels, value=labels[0])

interact(fair_dataset.show_bias_metrics, attr=attrs, label_value=labels)

Insight from previous graphics¶

Now you have a lot of informations about your dataset ! You can go deeper by transform your data and re-execute a notebook with this template if necessary.

Don't forget to detail your insight about this dataset on a worksheet or slides so that business people may understand what you found without going into this notebook by themselves

The end.¶

Thanks for reading. Nathan

	dataset	column	dtype	is_protected	new_name	description
0	adult	age	int64	1	NaN	Age of the person
1	adult	workclass	object	0	NaN	Workclass of the person (e.g. Private)
2	adult	fnlwgt	int64	0	final_weight	final weight, which is the number of units in ...
3	adult	education	object	0	NaN	Education level
4	adult	educational-num	int64	0	NaN	Education level (numerical format)
5	adult	marital-status	object	1	NaN	Marital status
6	adult	occupation	object	0	NaN	Field occupation
7	adult	relationship	object	0	NaN	Current relationship
8	adult	race	object	1	NaN	Ethnicity
9	adult	gender	object	1	NaN	Female or Male
10	adult	capital-gain	int64	0	NaN	Gain of capital
11	adult	capital-loss	int64	0	NaN	Loss of capital
12	adult	hours-per-week	int64	0	NaN	number of working hours during a week
13	adult	native-country	object	1	NaN	Native country
14	adult	income	object	0	NaN	Target : income > 50K or not

	age	workclass	final_weight	education	educational-num	marital-status	occupation	relationship	race	gender	capital-gain	hours-per-week	native-country	income
0	25	Private	226802	11th	7	Never-married	Machine-op-inspct	Own-child	Black	Male	0	40	United-States	<=50K
1	38	Private	89814	HS-grad	9	Married-civ-spouse	Farming-fishing	Husband	White	Male	0	50	United-States	<=50K
2	28	Local-gov	336951	Assoc-acdm	12	Married-civ-spouse	Protective-serv	Husband	White	Male	0	40	United-States	>50K
3	44	Private	160323	Some-college	10	Married-civ-spouse	Machine-op-inspct	Husband	Black	Male	7688	40	United-States	>50K
4	18	?	103497	Some-college	10	Never-married	?	Own-child	White	Female	0	30	United-States	<=50K

EDA on raw data (after collect)¶

Load packages¶

Load data informations¶

Load data using data informations¶

Display shape & head¶

adult, (48842, 15)¶

Set datasets into differents variables¶

Analyse : missing values¶

Missing values for adult dataset¶

Analyse : plot each variable¶

Using MLHelper¶

Numerical variables¶

Categorical variables¶

Datetime variables¶

Pearson correlation matrix for numerical variables¶

Cramers V correlation matrix for categorical variables¶

Point Biserial correlation matrix for numerical & categorical variables¶

Handle dataset bias¶

age¶

marital-status¶

race¶

gender¶

native-country¶

Using transparentai ClassificationDataset dataset¶

Insight from previous graphics¶

The end.¶